Skip to content

Conversation

@parkertimmins
Copy link
Contributor

@parkertimmins parkertimmins commented Feb 11, 2025

Fix test failures of the form All incoming requests on node [node_s1] should have finished. Expected 0 bytes for requests in-flight but got 298 bytes. A couple of fixes:

  • For test that adds doc without timestamp, then adds mapping with timestamp, ensure green between adding doc and adding mapping. This makes sure that doc has been written to all shards and thus that timestamp validation does not occur while doc is being written to a shard.
  • Delete pipeline in @Before, then wait for it to be re-created by the MigrateTemplateRegistry

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@elasticsearchmachine elasticsearchmachine added the Team:Data Management Meta label for data/management team label Feb 11, 2025
@parkertimmins parkertimmins changed the title Delete pipeline only in relevant tests rather than cleanup Fix ReindexDataStreamIndexAction timestamp validation bug in tests Feb 12, 2025
Copy link
Member

@masseyke masseyke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@parkertimmins parkertimmins added the auto-backport Automatically create backport pull requests when merged label Feb 12, 2025
@parkertimmins parkertimmins merged commit 05a2003 into elastic:main Feb 12, 2025
17 checks passed
@parkertimmins parkertimmins deleted the reindex-data-stream-fix-pipeline-second-try branch February 12, 2025 14:00
@elasticsearchmachine
Copy link
Collaborator

💔 Backport failed

Status Branch Result
9.0 Commit could not be cherrypicked due to conflicts
8.18 Commit could not be cherrypicked due to conflicts
8.x Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 122274

parkertimmins added a commit to parkertimmins/elasticsearch that referenced this pull request Feb 12, 2025
…lastic#122274)

Fix race condition test bugs related to the reindex-data-stream-pipeline. For tests that add doc without timestamp, then add mapping with timestamp, ensure green between adding doc and adding mapping. This makes sure that doc has been written to all shards and thus that timestamp validation does not occur while doc is being written to a shard. Delete pipeline in Before method, then wait for it to be re-created by the MigrateTemplateRegistry.
parkertimmins added a commit to parkertimmins/elasticsearch that referenced this pull request Feb 12, 2025
…lastic#122274)

Fix race condition test bugs related to the reindex-data-stream-pipeline. For tests that add doc without timestamp, then add mapping with timestamp, ensure green between adding doc and adding mapping. This makes sure that doc has been written to all shards and thus that timestamp validation does not occur while doc is being written to a shard. Delete pipeline in Before method, then wait for it to be re-created by the MigrateTemplateRegistry.

(cherry picked from commit 05a2003)

# Conflicts:
#	muted-tests.yml
#	x-pack/plugin/migrate/src/internalClusterTest/java/org/elasticsearch/xpack/migrate/action/ReindexDatastreamIndexTransportActionIT.java
@parkertimmins
Copy link
Contributor Author

💚 All backports created successfully

Status Branch Result
8.x
9.0
8.18

Questions ?

Please refer to the Backport tool documentation

parkertimmins added a commit to parkertimmins/elasticsearch that referenced this pull request Feb 12, 2025
…lastic#122274)

Fix race condition test bugs related to the reindex-data-stream-pipeline. For tests that add doc without timestamp, then add mapping with timestamp, ensure green between adding doc and adding mapping. This makes sure that doc has been written to all shards and thus that timestamp validation does not occur while doc is being written to a shard. Delete pipeline in Before method, then wait for it to be re-created by the MigrateTemplateRegistry.

(cherry picked from commit 05a2003)

# Conflicts:
#	muted-tests.yml
#	x-pack/plugin/migrate/src/internalClusterTest/java/org/elasticsearch/xpack/migrate/action/ReindexDatastreamIndexTransportActionIT.java
elasticsearchmachine pushed a commit that referenced this pull request Feb 12, 2025
…122274) (#122384)

Fix race condition test bugs related to the reindex-data-stream-pipeline. For tests that add doc without timestamp, then add mapping with timestamp, ensure green between adding doc and adding mapping. This makes sure that doc has been written to all shards and thus that timestamp validation does not occur while doc is being written to a shard. Delete pipeline in Before method, then wait for it to be re-created by the MigrateTemplateRegistry.
elasticsearchmachine pushed a commit that referenced this pull request Feb 12, 2025
…sts (#122274) (#122382)

* Fix ReindexDataStreamIndexAction timestamp validation bug in tests (#122274)

Fix race condition test bugs related to the reindex-data-stream-pipeline. For tests that add doc without timestamp, then add mapping with timestamp, ensure green between adding doc and adding mapping. This makes sure that doc has been written to all shards and thus that timestamp validation does not occur while doc is being written to a shard. Delete pipeline in Before method, then wait for it to be re-created by the MigrateTemplateRegistry.

(cherry picked from commit 05a2003)

# Conflicts:
#	muted-tests.yml
#	x-pack/plugin/migrate/src/internalClusterTest/java/org/elasticsearch/xpack/migrate/action/ReindexDatastreamIndexTransportActionIT.java

* getPipelines missing from 8x.
elasticsearchmachine pushed a commit that referenced this pull request Feb 12, 2025
…ests (#122274) (#122385)

* Fix ReindexDataStreamIndexAction timestamp validation bug in tests (#122274)

Fix race condition test bugs related to the reindex-data-stream-pipeline. For tests that add doc without timestamp, then add mapping with timestamp, ensure green between adding doc and adding mapping. This makes sure that doc has been written to all shards and thus that timestamp validation does not occur while doc is being written to a shard. Delete pipeline in Before method, then wait for it to be re-created by the MigrateTemplateRegistry.

(cherry picked from commit 05a2003)

# Conflicts:
#	muted-tests.yml
#	x-pack/plugin/migrate/src/internalClusterTest/java/org/elasticsearch/xpack/migrate/action/ReindexDatastreamIndexTransportActionIT.java

* getPipelines missing from 8.18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-backport Automatically create backport pull requests when merged backport pending :Data Management/Data streams Data streams and their lifecycles >non-issue Team:Data Management Meta label for data/management team v8.18.0 v8.19.0 v9.0.0 v9.1.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants